Video object clipping method and apparatus

ABSTRACT

A video object clipping method includes storing, in a storage unit, original images each including a video object to be clipped and reference alpha images representing objects prepared, determining a criteria original image and a criteria reference alpha image from the original images and the reference alpha images, determining a deformation parameter by deforming the criteria reference alpha image to correspond to the criteria original image, and deforming remaining ones of the reference alpha images according to the determined deformation parameter to generate output alpha images corresponding to the original images.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based upon and claims the benefit of priority fromthe prior Japanese Patent Application No. 2002-097760, filed Mar. 29,2002, the entire contents of which are incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates to a video object clipping method of clippingvideo objects from a video image and a video object clipping apparatus.

2. Description of the Related Art

There is a video object clipping technique for clipping a video objectfrom a video image. The technique is utilized for creating a catalog ora poster by combining the clipped video object with another backgroundimage, or for creating contents of a Web page.

In addition, the video object clipping technique can be used fordesignating an anchor area for producing such image contents thatprovide related information if a user points a video object with therelated information (name, a price).

As disclosed in a document “MPEG-4 standardized methods for thecompression of arbitrarily shaped video objects” N. Brady (IEEETransactions on Circuits and Systems for Video Technology, vol. 9, no.8, pp. 1170 1189, December 1999), MPEG-4 that is International Standardof video compression adopts an object encoding function for encodingdata every object. The video object clipping technique for clipping avideo object from a video image is used for the purpose of generatingobject data before encoding.

As a concrete example of the video object clipping technique there is“picture contour line extraction apparatuses” disclosed in Jpn. pat.Appln. KOKAI Publication No. 2001-14477. This is a method includinginputting a contour by drawing a line using a pointing device such as amouse, and modifying the contour. However, generally, a user must move apointing device along unevenness of the video object to be clipped fromthe video image, when inputting a contour by a manual input. As aresult, a time and labor are spent to input manually and accurately thecontour of the object.

On the other hand, in a clip tool of a writing brush printing softwaremade of Creo Co., Ltd., some templates are prepared beforehand, and adesired region is clipped by deforming the template (“Fude mame Ver. 11(Treadmark)” guidebook, page 109-110). If this tool is used, thetemplate can be used for clipping an object when the shape of the objectis similar to the template.

However, in a method of clipping a video object by deforming a referencealpha image, there is a problem that a user must adjust a deformationparameter by times corresponding to the number of the original images toclip the video object from a plurality of original images.

BRIEF SUMMARY OF THE INVENTION

It is an object of the present invention to provide a video objectclipping method and apparatus which generate an alpha image expressing avideo object from a plurality of original images.

According to an aspect of the invention, there is provided a method ofclipping an object image from an image comprising: storing, in a storageunit, a plurality of original images each including an object image tobe clipped and a plurality of reference alpha images representingobjects prepared beforehand; determining a criteria original image and acriteria reference alpha image from the original images and thereference alpha images; determining a deformation parameter by deformingthe criteria reference alpha image so as to correspond to the criteriaoriginal image; and deforming remaining ones of the reference alphaimages according to the determined deformation parameter to generateoutput alpha images corresponding to the original images.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING

FIG. 1 is a block diagram of a video object clipping apparatus accordingto a first embodiment of the present invention;

FIG. 2 is a flowchart which shows a flow of a first selection method ofselecting a criteria original image and a criteria reference alpha imageaccording to the first embodiment;

FIG. 3 is a flowchart which shows a flow of a second selection method ofselecting a criteria original image and a criteria reference alpha imageaccording to the first embodiment;

FIG. 4 is a diagram showing an image processing status determined by adeformation parameter determination unit 103 of the video objectclipping apparatus according to the first embodiment;

FIG. 5 is a diagram showing another image processing status determinedby the video object clipping apparatus according to the firstembodiment;

FIG. 6 is a flowchart showing a flow of a video object clipping processaccording to a second embodiment;

FIG. 7 is a diagram showing an example of clipping a moving object froma sequence of images according to a third embodiment;

FIG. 8 is a flowchart showing a flow of a video object clipping processaccording to a fifth embodiment;

FIG. 9 is a diagram showing a status of selectively determining acriteria reference alpha image in the video object clipping processaccording to the fourth embodiment;

FIG. 10 is a diagram showing a status of selectively determining acriteria reference alpha image in the video object clipping processaccording to the fourth embodiment;

FIG. 11 is a flowchart showing a flow of a video object clipping processaccording to a sixth embodiment;

FIG. 12 is a diagram showing a status of selectively determining acriteria reference alpha image in the video object clipping processaccording to the fifth embodiment;

FIG. 13 is a diagram showing a status of selectively determining acriteria reference image in the video object clipping process accordingto the fifth embodiment;

FIG. 14 is a diagram showing a status of skipping a reference alphaimage string in a video object clipping process according to a seventhembodiment;

FIG. 15 is a diagram showing a processing state of a video objectclipping apparatus according to the seventh embodiment;

FIG. 16 is a diagram showing a status of generating a plurality oftemporary reference alpha image string according to an eighthembodiment;

FIG. 17 is a diagram showing a status of generating a plurality ofoutput object alpha image strings according to a ninth embodiment;

FIG. 18 is a diagram showing a status of generating an output alphaimage string by the sum of a plurality of video objects in a videoobject clipping process according to a tenth embodiment;

FIG. 19 is a diagram showing an image processing status obtained by acontour correcting unit in a video object clipping process according toan eleventh embodiment;

FIG. 20 is a diagram showing an image processing status of a videoobject clipping apparatus according to the tenth embodiment;

FIG. 21 is a diagram showing GUI of a video object clipping apparatusaccording to a twelfth embodiment;

FIG. 22 is a flowchart showing a flow of GUI in a video object clippingprocess according to the twelfth embodiment;

FIGS. 23A and 23B are diagrams each showing a status of a combined imageobtained by a video object clipping apparatus according to a thirteenthembodiment;

FIG. 24 is a flowchart which shows flow of GUI in an object beginning tocut process of the thirteenth embodiment;

FIG. 25 is a diagram showing an image clipping process of a video objectclipping apparatus according to a fourteenth embodiment;

FIG. 26 is a diagram showing GUI of a video object clipping apparatusaccording to a fifteenth embodiment;

FIG. 27 is a flowchart showing a flow of GUI in the object clippingprocess according to the fourteenth embodiment;

FIG. 28 is a diagram showing an image clipping process of a video objectclipping apparatus according to a sixteenth embodiment; and

FIG. 29 is a diagram showing an image clipping process of a video objectclipping apparatus according to an seventeenth embodiment.

DETAILED DESCRIPTION OF THE INVENTION

There will now be described embodiments of the present invention inconjunction with the drawings.

First Embodiment

According to the first embodiment of FIG. 1, a video object clippingapparatus comprises a storage unit 101, a storage unit 102, adeformation parameter determination unit 103, an output alpha imagegeneration unit 104 and a storage unit 105. The storage unit 101 storesa plurality of original images in which a video object to be clipped iscontained. The storage unit 102 stores a plurality of reference alphaimages prepared beforehand. The deformation parameter determination unit103 determines a deformation parameter used for deforming a criteriareference alpha image corresponding to a criteria original image. Theoutput alpha image generation unit 104 generates output alpha images ofremaining original images by deforming remaining reference alpha imagesaccording to the deformation parameter. The storage unit 105 stores theoutput alpha images expressing video objects corresponding to originalimages 101 and generated by the output alpha image generation unit 104.A control unit 113 comprises a processor and the like, and is connectedto the above units 101 to 105 and a display unit 114 for controllingthese units.

The operation of the video object clipping apparatus will be describedreferring to FIG. 2.

A set of a criteria original image 106 and a criteria reference alphaimage 107 is selected from the storage unit 101 and the storage unit102. This may use a set of original images prepared as a criteriaoriginal image and a criteria reference alpha image beforehand, and areference alpha image.

If the original images and the alpha images correspond to each other,the set of original image and reference alpha image is displayed on thedisplay unit 114 every set of images. A user determines a criteriaoriginal image and a criteria reference alpha image on the basis of thedisplayed images.

In other words, at first, the set i of the original image and referencealpha image are reset (S11). The image set i is compared with the numberof the images (S12). If i<the number of images, the original image andreference alpha image of the set i are displayed on the display unit 114(S13). If the displayed criteria original image and criteria referencealpha image match with each other, the user instructs a determination(S14). However, if they do not match with each other, the user does notinstruct the determination. If the determination is not instructed, i isincremented (S15), and the process returns to step 12. This process isrepeated.

When the determination is instructed in step S14, the criteria originalimage and criteria reference alpha image are decided, and the process isfinished. If i exceeds the number of images in step S12, the processreturns to step 11 and i is reset.

If the original images and the reference alpha images do not correspondto each other, the original images are displayed sequentially, and theuser determines a criteria original image from the displayed images. Thereference alpha images are displayed sequentially together with thedetermined criteria original image. The user determines a criteriareference alpha image from the sequentially displayed reference alphaimages. As a result, the criteria original image and criteria alphaimage are decided.

In other words, as shown in FIG. 3, at first i is reset (S111) and i<thenumber of original images is determined (S112). If the determination instep 112 is YES, the i-th original image is displayed (S113). It isdetermined whether the user instructs a determination based on thisdisplayed image (S114). When this determination is NO, the process isincremented (S115) and returns to step S112. When the determination isYES, the criteria original image is decided (S116). Then, i is reset(S117) and it is determined whether i is smaller than the number ofreference alpha images (S118). When this determination is YES, thecriteria original image and i-th reference alpha image are displayed(S119). It is determined whether the determination of the referencealpha image is instructed by the user (S120). When this determination isNO, i is incremented (S121), and the process returns to step S118. Whenthe determination is YES, the criteria alpha image is determined (S122)and the process is finished.

The selected criteria original image and criteria reference alpha imageare input to the deformation parameter decision unit 103. Thedeformation parameter determination unit 103 generates an output alphaimage 108 corresponding to the criteria original image 106 as shown inFIG. 4, when a criteria reference alpha image 107 corresponding to thecriteria original image 106 is deformed according to a certaindeformation parameter. The output alpha image 108 is stored in theoutput alpha image storage unit 105. The deformation parameter may bedetermined by displaying the criteria reference alpha image 107 and thecriteria original image 106 by superposing one on another, dragging theimages by means of a pointing device such as a mouse to move the imagesin parallel, changing the size of the image or rotating the image.Alternatively, the criteria reference alpha image 107 may be deformed byinput of a deformation parameter value. The deformation parameterdetermination unit 103 outputs a deformation parameter 109 indicating arelation between the criteria original image 106 and criteria referencealpha image 107 and the output alpha image 108 of an original image.

Remaining reference alpha images 111, remaining original images 110 andthe deformation parameter 109 determined by the deformation parameterdetermination unit 103 are input to the output alpha image generationunit 104. The output alpha image generation unit 104 deforms theremaining reference alpha images 111 by the deformation parameter 109determined by the deformation parameter determination unit 103 as shownin FIG. 5. As a result, the output alpha images 112 of the remainingoriginal images 110 are generated and stored in the output alpha imagestorage unit 105.

By the above embodiment, the output alpha images expressing videoobjects can be easily generated with respect to a plurality of originalimages.

Second Embodiment

FIG. 6 is a flowchart showing a flow of a video object clipping methodaccording to the second embodiment of the present invention. In thevideo object clipping method, at first a criteria original image and acriteria reference alpha image are selected from a plurality of originalimages and a plurality of reference alpha images, similarly to the firstembodiment (S211). The criteria original image and criteria referencealpha image are displayed on the display unit (S212). In this display,these images may be displayed on separated windows. Also, the criteriareference alpha image may be made semitransparent and superposed on thecriteria original image.

It is determined whether the criteria reference alpha image clips thevideo object of the criteria original image (S213). When thisdetermination is NO, the deformation parameter is adjusted (S214), andthe criteria reference alpha image is deformed according to the adjusteddeformation parameter (S215). The process returns to step S212.

In step S213, when the deformed criteria reference alpha image clips thevideo object of the criteria original image, this deformed criteriareference alpha image is output as an output alpha corresponding to thecriteria original image (S216), and the deformation parameter isdetermined (S217).

Using the same deformation parameter as the deformation parameterobtained as described above, all remaining reference alpha images aredeformed (S218). As a result, the deformed reference alpha images areoutput as a plurality of output alpha images corresponding to aplurality of original images (S219).

By the embodiment, the output alpha images expressing video objects canbe easily generated with respect to a plurality of original images.

Third Embodiment

In this embodiment, the original images stored in the storage unit 101are not a plurality of original images corresponding to differentobjects, but a string of sequential images obtained by imaging the sameobject. Also, the reference alpha images stored in the storage unit 102are not a plurality of object templates corresponding to differentobjects, but a string of reference alpha images expressing the templateof the same object. Accordingly, a moving object can be clipped from thevideo image.

There will now be described an operation for clipping a video objectfrom a string of images obtained by deforming a string of referencealpha images, referring to FIG. 7. The sequential images in FIG. 7 showa status that a car moving from the right to the left in a screen. Thestring of reference alpha images wherein the car moves from the right tothe left on a screen is prepared as a string of reference alpha images.For example, an original image of the first frame and a reference alphaimage thereof are selected as a set of a criteria sequential image and acriteria reference alpha image. The set of criteria original image andcriteria reference alpha image are input to the deformation parameterdetermination unit 103, to determine an output alpha image of the firstframe and a deformation parameter. The output alpha image generationunit 104 generates a string of output alpha images corresponding toremaining sequential images by deforming remaining reference alphaimages according to the deformation parameter.

According to the above embodiment, the moving video object can beclipped from the sequential image using the deformation of the referencealpha image. The present method is applied not only to clipping the carthat moves in the same direction, but also to extracting an object suchas a person, from a sequential image which can prepare a string ofreference alpha images, for example, a sports image such as a serve of atennis, a throw of a baseball or a swing of a golf that movement of theobject is approximately constant every time.

Fourth Embodiment

In this embodiment, the deformation parameter 109 is limited todeformations such as a parallel movement, a size conversion and arotation. As a result, the determination and expression of thedeformation parameter are simplified. The deformation based on theparallel movement, a size conversion, a rotation, a reverse, etc. iscalled an affine deformation. This is a deformation expressed by thefollowing equation: $\begin{pmatrix}X \\Y\end{pmatrix} = {{\begin{pmatrix}a & b \\c & d\end{pmatrix}\begin{pmatrix}x \\y\end{pmatrix}} + \begin{pmatrix}e \\f\end{pmatrix}}$

The deformation parameter is expressed by a parameter (a, b, c, d) onthe scaling and rotation, and a parameter (e, f) on the parallelmovement. This deformation parameter can be determined by a process ofdisplaying an original image and a reference alpha image by superposingone on another, dragging the superposed image by a pointing device suchas a mouse, and subjecting the image to a scaling, a rotation and aparallel movement. The deformation parameter can be determined bydirectly inputting a deformation parameter value representing thescaling, rotation and parallel movement.

By the above embodiment, a designated object can be easily clipped froma plurality of original images using an affine deformation parameter.

Fifth Embodiment

In this embodiment, the display unit 113 displays a criteria originalimage and displays selectively a plurality of reference alpha images.One of the displayed reference alpha images is determined as a criteriareference alpha image by the user. This configuration permits to selecta criteria reference alpha image from the reference alpha images.

Using a flowchart shown in FIG. 8, a method of making a sequential imageassociate with a string of the reference alpha images with respect to atime axis will be explained. The user selects a frame of a criteriasequential image from the sequential images (S311). This image may use asequential image prepared beforehand as the criteria sequential image,and may determine the criteria original image by a method of determininga criteria image that is shown in FIG. 3.

i is reset (S312) and it is determined whether i is not less than 0(S313). If this determination is YES, it is determined whether i is lessthan the number of all frames (S314). If this determination is NO, 1 isadded to all frames and the process advances to step S316. If thedetermination is YES, the process advances to step 316. In step 316, theselected criteria sequential image and the i-th reference alpha imagethat is a temporary criteria reference alpha image are displayed. Theseimages may be displayed on separated windows. As shown in FIG. 9, acomposite image obtained by superposing the criteria sequential imageand the i-th reference alpha image may be displayed. The i-th referenceimage is a temporary criteria reference alpha image displayedselectively.

The user watches the criteria sequential image and temporary criteriareference alpha image displayed selectively, and determines whether thei-th reference alpha image which now is displayed as shown in FIG. 10 issuitable as a criteria reference alpha image. When this determination isYES, the reference alpha image is determined as a criteria referencealpha image (S319). If the determination is NO, the user watches thecriteria sequential image and temporary criteria reference alpha imagedisplayed selectively and determines whether the suitable criteriareference alpha image is a past frame or a future frame in time than thecurrently displayed the i-th reference alpha image. Then, the temporarycriteria reference alpha image is shifted backward and forward in thetime axis as shown in FIG. 9 (S318). Thereafter, the process returns tothe process of displaying the temporary criteria reference alpha imageshifted in the time axis and the criteria sequential image.

The frame number may be changed by a button for displaying a past frameor a future frame of the string of reference alpha images.Alternatively, the frame number may be changed by designating anarbitrary frame number with a slide bar. This process is repeated tillthe suitable criteria reference alpha image is determined. As thusdescribed, the string of temporary criteria reference alpha images isadjusted in the time axis. When the criteria reference alpha image thatmatches with the criteria sequential image in phase as shown in FIG. 10was found, the criteria sequential image is associated with the criteriareference alpha image. Simultaneously, each of the remaining sequentialimages is associated with the sequence of reference alpha images. Also,the criteria sequential image fixed till now so as not to be updated anddisplayed in a frame number changing process in the time axis isreleased. Thereafter, the sequential image and reference alpha imagethat associate with each other are displayed while being updated by theframe number changing process.

According to the above embodiment, the association of the sequentialimage with the reference alpha image can be made in the time axis. Afterthis process, it is possible by the method explained in the thirdembodiment to perform a process for clipping a video object in a videoimage, that is, in a spatial area, to clip a moving object in the image.

Sixth Embodiment

In this embodiment, the display unit 113 displays a criteria referencealpha image and displays selectively a plurality of original images. Oneof the original images is determined as a criteria original image by theuser. This configuration permits to select a criteria original imagecorresponding to the criteria reference alpha image.

Using a flowchart shown in FIG. 11, a method of making the sequentialimages associate with the string of reference alpha images with respectto a time axis will be explained. A user selects one frame of criteriareference alpha image from the string of reference alpha images (S411).This may use a reference alpha image prepared beforehand as a criteriareference alpha image, and may be determined by a method of determininga criteria original image in FIG. 3.

i is reset to 0 (S412) and it is determined whether i is less than thenumber of all frames (S414). When this determination is YES, the processadvances to step S416, If the determination is NO, 1 is added to thenumber of all frames and the process advances to step 416. In step 416,the selected criteria reference alpha image and i-th sequential imagewhich is a temporary criteria sequential image are displayed. Bothimages may be displayed on separated windows side by side. The criteriaalpha image and the i-th sequential image which is a temporary criteriasequential image is selectively displayed may be displayed as acomposite image as shown in FIG. 12.

The user watches the criteria reference alpha image and the temporarycriteria sequential image displayed selectively, and determines whetherthe i-th sequential image which is presently displayed as shown in FIG.13 is suitable for a criteria sequential image (S417). If thisdetermination is YES, the sequential image is determined as a criteriasequential image (S319). If the determination is NO, the user watchesthe criteria reference alpha image and the temporary criteria sequentialimage selectively displayed and determines whether the suitable criteriasequential image is a past frame or a future frame in time than the i-thsequential image which is currently displayed. The temporary criteriasequential image is shifted backward and forward in the time axis asshown in FIG. 12. The process returns to a step of displaying thecriteria reference alpha image and the temporary criteria sequentialimage whose frame is shifted in a time axis (S318).

The frame number may be changed every one frame by operating a buttonfor making a display unit display a past frame or a future frame of thesequential image, or by designating an arbitrary frame number with aslide bar. This process is repeated till a suitable criteria sequentialimage is determined.

As thus described, when the criteria sequential image which matches withthe criteria reference alpha image in phase as shown in FIG. 13 is foundby adjusting the temporary criteria sequential image in a time axis, thecriteria reference alpha image and the criteria sequential image areassociated with each other. Simultaneously, the string of the remainingreference alpha images is associated with the sequential image. Also,the criteria sequential image fixed till now so as not to be updated anddisplayed by a frame number changing process in a time axis is releasedin fixing. Thereafter, the sequential image and the reference alphaimage that are associated with each other are displayed while beingupdated by the frame number changing process.

According to the above embodiment, the association of the sequentialimage with the reference alpha image can be made in a time axis. Afterthis process, it is possible by the method explained in the thirdembodiment to perform a process for clipping a video object in an image,that is, in a spatial area, to clip a moving object in the image.

The Seventh Embodiment

In this embodiment, a string of a plurality of reference alpha images,i.e., a reference alpha image string that is prepared beforehand has ahigher frame rate in a time axis than the sequential image. A set ofcriteria sequential image and criteria reference alpha image isdetermined by a method that explained in the fifth and sixth embodimentsor an image processing method. The sequential image and the referencealpha image string make the same time frame rate to be in phase bydownsampling (skipping or thinning) the reference alpha image sequenceusing the determined criteria reference alpha image as a criteria. Astring of output alpha image strings, i.e., an output alpha image stringis generated using the reference alpha image string whose time framerate is the same as that of the sequential image. As a result, it ispossible to clip a moving object from the sequential image.

In FIG. 14, the reference alpha image string is prepared at a frame ratetwo times in the time axis with respect to the sequential image. A setof criteria original image and criteria reference alpha image thatassociate with each other is found from the sequential image and thereference alpha image string. This can be found by the methods explainedin the fourth and fifth embodiments or automatically by an imageprocessing method.

As shown by an arrow (a) in FIG. 14, a reference alpha imagecorresponding to the criteria original image of the first frame isselected. In this embodiment, since the reference alpha image string isprepared at a frame rate two times that of the sequential image, thereference alpha images corresponding to the remaining sequential imagesare determined as shown by an arrow (b) in FIG. 14 by selecting thereference alpha images with one frame being skipped. As a result, thesequential image and the reference alpha image string are the same timeframe rate to be in phase. The subsequent process is the same as thethird embodiment.

There will now be described a method for preparing a reference alphaimage string at a high frame rate and clipping a video object from asequential image using deformation of the reference alpha image string,referring to FIG. 15.

The criteria sequential image and criteria reference alpha image thatare associated as shown by the arrow (a) in FIG. 14 are input to thedeformation parameter determination unit 103, to determine an outputalpha image and a deformation parameter with respect to the criteriasequential image. The output alpha image generation unit 104 deforms thereference alpha images that are associated with the remaining sequentialimages as shown by the arrow (b) in FIG. 14 by the same deformationexpressed by the deformation parameter. As a result, the output alphaimages corresponding to the remaining sequential images can begenerated.

The frame rate of the reference alpha image string which is preparedbeforehand in the present method is not limited to an integer time thatof the sequential image, but may be an arbitrary time. In addition, itis not necessary to prepare a frame rate increased at a constant rate ina time axis.

According to the above embodiment, even if the reference alpha imagestring has a higher frame rate in a time axis than the sequential image,it is possible to clip a desired video object from the sequential image.This enables to make the captured sequential image associate with thestring of reference alpha images whose phase is near in time to thesequential image. As a result, it is possible to extract a moving objectwith a good precision.

Eighth Embodiment

According to the embodiment, a string of a plurality of reference alphaimages, i.e., a reference alpha image string that is prepared beforehandhas a higher frame rate in a time axis than that of the sequentialimage. A string of temporary reference alpha images with a plurality ofphases is generated by downsampling the reference alpha image stringusing a plurality of temporary criteria alpha images as a criteria. Aplurality of temporary output alpha image strings are generated by thetemporary reference alpha image strings with the plural phases. At ahigh-speed and a good precision can be clipped a moving object from thesequential image by selecting one of the plurality of temporary outputalpha image strings as a final output alpha image string.

In FIG. 16, the reference alpha image string is prepared at a frame ratetwo times that of the sequential image in a time axis. Since the framerate is two times, if the reference alpha image is selected with oneframe being skipped as shown in FIG. 16, the temporary reference alphaimage strings 1 and 2 which are the same in a frame rate and shifted ina phase can be generated beforehand. The temporary output alpha imagestrings are generated based on the temporary reference alpha imagestrings 1 and 2. The final output alpha image string is selected fromtwo temporary output alpha image strings.

According to the above embodiment, even if the reference alpha imagestring has a higher frame rate than that of the sequential image in atime axis, it is possible to clip a desired video object from thesequential image with a higher speed and a good precision.

Ninth Embodiment

In the present embodiment, a plurality of reference alpha image stringsare prepared for a plurality of objects individually. A plurality ofdeformation parameters are determined individually by deforming thecriteria reference alpha images of the reference alpha image string. Thecriteria reference alpha images of the remaining reference alpha imagestrings are individually deformed by the deformation parameters, togenerate a plurality of strings of output alpha images. As a result, itis possible to clip a plurality of moving objects from a video image.

In FIG. 17, two reference alpha image strings 1 and 2 that represent twokinds of objects are prepared. In other words, a template wherein a carmoves from the right to the left is prepared as the reference alphaimage string 1, and a template wherein an airplane moves from the rightto the left is prepared as the reference alpha image string 2. When thesequential images shown in FIG. 17 are input, respective video objectsare subjected independently to the moving object clipping process whichis described in the third embodiment. As a result, it is possible togenerate an output alpha image string 1 and an output alpha image string2 as shown in FIG. 17.

Tenth Embodiment

According to this embodiment, the video objects of the output alphaimage strings provided by the moving object clipping apparatus of theninth embodiment are added to generate a string of composite outputalpha images. In FIG. 18, the moving objects are clipped independentlyby the reference alpha image strings 1 and 2. Thereafter, the outputalpha image strings are combined to generate the composite output alphaimage string.

Eleventh Embodiment

This embodiment performs a postprocessing of the moving object clippingprocess in the first embodiment or the moving object clipping processfor clipping a moving object from the sequential image in the thirdembodiment. In other words, the position of the contour which is aboundary between an video object and a background region in the outputalpha image is corrected every frame for a plurality of output alphaimages or an output alpha image string. As a result, the precision ofthe object clipping can be improved. The correction of the contour usesluminance information of the original image and contour information ofthe output alpha image. Using technique such as a literature “Highaccuracy detection of subject contour with the use of LIFS”, Ida,Sanbonnsugi, Watanabe (Institute of Electronics, Information andCommunication Engineers article magazine, D-II, Vol, J82-D-II, NO. 8,page 1282-1289, August, 1999), the position of contour can be corrected.This contour correction is shown in FIG. 19.

When a rough contour as shown in black is drew to a bear as shown in theleft of FIG. 19, the contour is corrected as shown in the right of FIG.19. As a result, the clipping precision increases. When a plurality ofgenerated output alpha images and a plurality of original images areinput to the contour correction unit as shown in FIG. 20, a plurality ofcorrected output alpha images can be generated. In FIG. 20, the videoobjects of a plurality of remaining original images are clipped usingthe deformation of a plurality of remaining reference alpha imagesaccording to a deformation parameter indicating a deformation of thecriteria original image and criteria reference alpha image similarly tothe method of the first embodiment. Each of a plurality of output alphaimages is input to the contour correction unit. Also, the originalimages corresponding to the output alpha images are input to the contourcorrection unit. The video object expressed by these output alpha imagesis defined as a schematic contour of the object. The schematic contouris corrected by luminance information of the schematic contour andoriginal image. This process is performed every frame and a plurality ofcorrected output alpha images corresponding to a plurality of originalimage are finally output.

By the above embodiment, the video object can be clipped at a goodprecision.

Twelfth Embodiment

GUI (Graphical User Interface) for outputting, as output alpha images,alpha images expressing the video objects of a plurality of originalimages is explained hereinafter.

FIG. 21 shows an example of GUI for adjusting a string of referencealpha images with respect to a time axis and a spatial axis. A button2101 is used for determining or releasing a criteria sequential image tobe fixed in a time axis to match a phase between the sequential imageand the reference alpha image string as described in the fifthembodiment. A button 2102 is used for determining or releasing acriteria reference alpha image to be fixed in a time axis to match aphase between the sequential image and the reference alpha image stringas described in the sixth embodiment. A user selects a suitable criteriasequential image and criteria reference alpha image using buttons 2103and 2104. The button 2103 is used for displaying a temporary criteriareference alpha image which is not fixed or a temporary criteriasequential image with one frame changed to a past frame in time. Thebutton 2104 is used for displaying a temporary criteria reference alphaimage which is not fixed or a temporary criteria sequential image withone frame changed to a future frame in time. When the buttons 2101 and2102 are a release mode together, the sequential image and the referencealpha image string are changed in a time axis. If the criteriasequential image and the criteria reference alpha image are associatedin the time axis, the spatial positions and sizes of the criteriasequential image and criteria reference alpha image are matched by thebuttons 2105, 2106 and 2107. The button 2105 performs a scaling of thereference alpha image. The button 2106 performs a parallel movement ofthe reference alpha image. The button 2107 is actuated for rotating thereference alpha image. The actuations of the buttons 2101 to 2107 supplycontrol signals to the control unit 113 so that the desired processesare executed by the control unit 113.

The above-mentioned adjustment process for the time axis and spatialaxis is displayed on a screen 2108 to ensure a deformation result. Inthe above embodiment, the original image and reference alpha image aresuperposed and displayed, but may be displayed on separated windows.

The adjustment of the sequential image and the reference alpha imagestring with respect to the time axis and spatial axis with the use ofthe GUI will be described referring to FIG. 22. At first, a currentlyselected sequential image (i-th frame) and reference alpha image (j-thframe) are displayed on a display screen 2108 (S511).

It is determined whether the past button 2103 is set to a past frame(S512). If this determination is YES, it is determined whether the imagebutton 2101 is a fixing mode or a release mode (S513). If the releasemode is set, the frame number of the sequential image is updated to a(i−1)-th frame (S514). If the image button 2101 is set to the fixingmode, the frame number of the sequential image is fixed to the i-thframe.

It is determined whether the alpha image string button 2102 is set tothe fixing mode (S515). If the button 2102 is set to the release mode,the frame number of the reference alpha image is updated to the (j−1)-thframe (S516). If the button 2102 is set to the fixing mode, the framenumber of the reference alpha image is fixed to the j-th frame, and theprocess returns to step S511. In other words, the sequential image (i-thframe) and the reference alpha image (j-th frame) are displayed on thedisplay screen 2108.

When the determination in step S512 is NO, whether the future button2104 is set by a future frame (S517) is determined. When thisdetermination is YES, it is determined whether the image button 2101 isset to the fixing mode or the release mode (S518). When the image button2102 is set to the release mode, the frame number of the sequentialimage (i+1) is updated to a frame (S519). When carbon button 2101 is setto the fixing mode, the frame number of the sequential image is fixed tothe i-th frame. It is determined whether the alpha button 2102 is set tothe fixing mode or the release mode (S520). When the alpha button 2102is set to the release mode (S520), the frame number of the referencealpha image is updated to (j+1)-th frame (S521). When the alpha button2102 is set to the fixing mode, the frame number of the reference alphaimage is fixes to the j-th frame. In other words, the frame numbers iand j are updated according to the status of each button, the processreturns to the process to make the display screen 2108 display thesequential image (i-th frame) and the reference alpha image (j-th frame)(S511).

By the above-described flow, the phase matching between the sequentialimage and the reference alpha image string that is explained in thefifth to eighth embodiments can be realized. A process of adjusting thesequential image and reference alpha image string in a spatial axis willbe described referring to FIG. 22.

The criteria sequential image and criteria reference alpha image thatare adjusted in a time axis are displayed on the display screen 2108.Whether the scaling mode is set by the scaling button 2105 is determined(S522). When this determination is YES, a part showing a video object inthe criteria reference alpha image is scaled up or scaled down (S523).When the determination of the scaling mode is NO, whether a parallelmovement mode is set by a parallel movement button 2106 is determined(S524). When this determination is YES, the part showing the videoobject in the criteria reference alpha image is moved in parallel(S525). When the determination of the parallel movement mode is NO,whether a rotation mode is set by a rotation button 2107 is determined(S526). When this determination is YES, the part showing the videoobject in the criteria reference alpha image is rotated (S527). Theprocess returns to step S511 to display the deformed criteria referencealpha image as a new criteria reference alpha image.

By the above-described flow, the spatial adjustment of the sequentialimage and the reference alpha image string that is explained in thefirst to fourth embodiments and the ninth to eleventh embodiments can beexecuted. As mentioned above, by using GUI, the user can generate aplurality of output alpha images from a plurality of original images.

Thirteenth Embodiment

Using a plurality of output alpha images obtained by the object clippingmethod explained in the first embodiment, a new imaging effect can beobtained by superposing and displaying clipped video objects. Theimaging effect is shown in FIGS. 23A and 23B. FIGS. 23A and 23B showexamples superposing and displaying the clipped video images of themoving cars explained in the third embodiment. The process of combiningsequential images will be described referring to a flowchart of FIG. 24.

At first, a first frame sequential image is output as a first frame ofcomposite sequential image (S611). The images from the second frame tothe last N-th frame are combined. In this case, i is set to 2 (S612). Itis determined that i<the number N of all frames (S613). When thisdetermination is YES, a plurality of video objects are clipped from aplurality of output alpha images in the first embodiment (S614). A videoobject is clipped from the sequential image of the i-th frame using theoutput alpha image of the i-th frame (S615). The clipped video object ofthe i-th frame sequential image is superposed on the (i−1)-th framecomposite image (S615). This composite image is output as the i-th framesequence composite image (S616). Thereafter, i is updated and theprocess returns to step S613 (S617). The above steps are executed tillN-th frame to generate a new sequence composite image. In other words,the composite image of 1st to N-th frames is output as a new sequencecomposite image (S618).

As thus described, in the sequential image, when the image of the firstframe, the complex image of the first and second frames, the compositeimage of the first to third frames . . . the composite image of thefirst to N-th frames are displayed as new sequential images in turn, thelocus of a car can be watched with movement. The locus of the car iswatched as shown in FIG. 23A, and the display image can be uses as a newimaging effect.

FIG. 23A shows the locus that a car runs linearly. When the carapproaches an imaging point and then goes away, depth informationenables to generate a special effect image with realistic sensations asshown in FIG. 23B. In this case, if the depth information is given tothe clipped video object approaching more the imaging position, thewhole of the object is visible. In contrast, when the object goes awayfrom the imaging position, large depth information is given to thecutout object. As a result, there is generated such a composite imagerepresenting a perspective configuration that other clipped videoobjects are superposed sequentially on the rear side of the clippedvideo object most closer to the imaging position as shown in FIG. 23B.The special effect image with high realistic sensations can be realizedby giving the depth information to the clipped video object informationaccording to the far and near distances of the moving object. As amodification, in step S615 shown in FIG. 24 may be provided steps ofdetermining the far and near relation between the clipped video objectsbefore superposing the clipped video objects and giving the depthinformation to the clipped video objects according to the far and nearrelation of the clipped video objects. The above processes can beexecuted by a computer program.

If such a technique is applied to sports image and so on, it can beapplied to a form analysis, an effective instructor image, etc.

Fourteenth Embodiment

In the present embodiment, a marking is attached to a reference alphaimage string beforehand. The criteria reference alpha image andreference sequential image that are nearest in time are found by givingmarking to the object of a frame in a sequential image. The phases ofthe found images are matched. In FIG. 25, in the reference alpha imagestring that a car moves from the right to the left in a screen, amarking is attached to the vicinity of the center of a car bodybeforehand. A reference sequential image is selected. When a criteriareference alpha image corresponding to the selected reference sequentialimage is to be obtained, a marking is attached to an object in thereference sequential image.

A marking ◯ is attached to the vicinity of the center of the car body inthe reference sequential image shown in FIG. 25. If the reference alphaimage having the marking ◯ in the location that is nearest to themarking ◯ in the reference sequential image is searched by calculating adistance between the coordinate positions of the markings ◯, thecriteria reference alpha image corresponding to the reference sequentialimage can be determined automatically.

Fifteenth Embodiment

This embodiment is a modification of a method of matching the phaseexplained in the fourteenth embodiment. In this embodiments, a displayscreen 2601 displays a criteria sequential image selected fromsequential images, and a marking unit 2602 which inputs the marking ◯ tothe displayed criteria sequential image to perform a phase matching. Inother words, the display screen 2601 that displays the selectedreference sequential image and the marking unit 2602 which inputs themark of the displayed criteria sequential image are provided as shown inFIG. 26. The marking unit 2602 uses a pen tablet, but may use a mouse.

The phase matching will be described using a flow chart shown in FIG.27. At first, a reference sequential image is selected (S711) and it isdisplayed on the display screen 2601 (S712). A user indicates a videoobject in the reference sequential image using the marking unit 2602 andinputs a marking position (S713). Where of the object the user indicatesis different by the position of the marking added to the reference alphaimage string beforehand. The indication point is, for example, “thecenter of the car body” as shown in FIG. 26, or “the leftmost edge ofthe car body”.

The reference alpha image having a marking that is nearest in positionto the input marking is searched from the reference alpha image stringas explained in the fourteenth embodiment (S714). The searched referencealpha image is determined as the criteria reference alpha image (S715).

As thus described, by determining the criteria reference alpha imagecorresponding to the reference sequential image automatically, the phasematching between the sequential image and reference alpha image sequencebecomes possible. The above process is executed by the control unit 113shown in FIG. 1 in association with the marking unit 2602.

Sixteenth Embodiment

The present embodiment is a modification of the moving object clippingapparatus of the third embodiment. In the present embodiments, thebackground image of the sequential image is saved beforehand. Theposition of a video object is detected by calculating a differencebetween the background image and the criteria sequential image. Thereference alpha image nearest in time to the reference sequential imageis detected based on the detected position, whereby the phase matchingcan be performed.

In FIG. 28, one frame of the sequential image showing that a car movesfrom the right to the left in the screen is selected as the referencesequential image. The background image in which the car as the object isnot imaged is prepared beforehand. The error of each pixel value of thereference sequential image and the background image is calculated as anabsolute error or an average square error. By subjecting the error to athreshold process is generated a differential image shown in FIG. 28.This differential image has a pixel value “1” in an object part and apixel value “0” in a background part, for example, and indicates thesame configuration as the reference alpha image and the output alphaimage which mentioned above. A criteria reference alpha image mostsimilar to the differential image is searched from the reference alphaimage string. The criteria reference alpha image can be found bysearching for a reference alpha image whose video object overlaps withthat of a differential image most greatly or searching for a referencealpha image whose video object has the center of gravity which isnearest to that of the video object of a differential image. In FIG. 28,the criteria reference alpha image is selected automatically as shown by(a).

According to the above embodiment, the criteria reference alpha imagecorresponding to the reference sequential image can be found, wherebythe phases of the reference alpha image sequence and the sequentialimage are matched automatically.

Seventeenth Embodiment

The present embodiment is a modification of the moving object clippingapparatus of the third present embodiment. The moving object clippingapparatus is provided with a first differential image generation unitwhich generates a first differential image between a criteria sequentialimage of sequential images and an image of a past frame, a seconddifferential image generation unit which generates a second differentialimage between the criteria reference sequential image and an image of afuture frame, a detection unit which detects a position of an object bygenerating a logical product image obtained by logically multiplying thepixel values of the first differential image and second differentialimage, and a phase matching unit which matches the criteria sequentialimage and a reference alpha image in phase by finding the referencealpha image nearest to the criteria sequential image with respect to atime, thereby to enable a phase matching.

In FIG. 29, the n-th frame of the sequential image indicating a carmoving from the right to the left in a screen is selected as a referencesequential image. The error of each pixel value of the n-th framecriteria sequential image and the (n−1)-th frame sequential image arecalculated as an absolute error or an average square error. Bysubjecting the error to a threshold process is generated a differentialimage shown in FIG. 28. This differential image has a pixel value “1” inan object part and a pixel value “0” in a background part, for example,and indicates the same configuration as the reference alpha image andthe output alpha image which mentioned above.

The error between pixel values of the n-th frame criteria sequentialimage and the (n+1)-th frame sequential image is calculated, and adifferential image B between the n-th frame criteria sequential imageand (n+1)-th frame as shown in FIG. 29 is generated by subjecting theerror to a threshold process. The pixel values of the differential imageA and differential image B is subjected to a logical multiplication togenerate a logical multiplication image of the differential images A andB. A part of the video object in the n-th frame reference sequentialimage is included in this logical multiplication image. The criteriareference alpha image most similar to this logical multiplication imageis searched from the reference alpha image string. The criteriareference alpha image can be found by searching for a reference alphaimage whose video object overlaps with that of a differential image mostgreatly or searching for a reference alpha image whose video object hasthe center of gravity which is nearest to that of the video object of adifferential image. In FIG. 29, the criteria reference alpha image isselected automatically as shown by (a).

According to the above embodiment, the criteria reference alpha imagecorresponding to the reference sequential image can be found, wherebythe phases of the reference alpha image sequence and the sequentialimage are matched automatically.

As described above, according to the video object clipping apparatus ofthe present invention, a plurality of alpha images are prepared asreference alpha images beforehand. A criteria reference alpha imagecorresponding to a reference original image is deformed to generate anoutput alpha image of the original image. The remaining reference alphaimages are deformed according to a deformation parameter used for theabove deformation to generate output alpha images of the remainingoriginal images. As a result, alpha images expressing video objects of aplurality of original images can be easily generated. The processesaccording to the first to seventeenth embodiments can be executedsubstantially by the control unit 113.

Additional advantages and modifications will readily occur to thoseskilled in the art. Therefore, the invention in its broader aspects isnot limited to the specific details and representative embodiments shownand described herein. Accordingly, various modifications may be madewithout departing from the spirit or scope of the general inventiveconcept as defined by the appended claims and their equivalents.

1-25. (canceled)
 26. A video object clipping method comprising: storing,in a storage unit, a plurality of sequential original imagesrepresenting a substantially constant movement of a to-be-clipped objectand a plurality of prepared reference alpha images representing aplurality of sequential reference images representing a substantiallyconstant movement of a reference object; determining a criteria originalimage and a criteria reference alpha image from the original images andthe reference alpha images; determining a deformation parameter bydeforming the criteria reference alpha image to correspond to thecriteria original image, the deformation parameter representing at leastone of a parallel movement of the original object, a scaling of theoriginal object and a rotation of the original object; and deformingremaining ones of the reference alpha images according to the determineddeformation parameter to generate output alpha images corresponding tothe original images, wherein determining the deformation parameterincludes determining the deformation parameter based on matching of thecriteria original image with the criteria reference alpha image, whereinthe method further includes generating a string of the reference alphaimages at a frame rate in a time axis, determining a set of the criteriaoriginal image and the criteria reference alpha image, performing aphase matching by equalizing a frame rate between a string of thesequential original images and the string of the reference alpha images,the frame rate being equalized by downsampling the string of thereference alpha images using the determined criteria alpha image, andgenerating the output alpha images using the string of reference alphaimages having the same frame rate as that of the string of thesequential original images.
 27. The method according to claim 26,wherein determining the criteria original image and the criteriareference alpha image includes determining one of the plurality ofreference alpha images as the criteria reference alpha image bydisplaying selectively the plurality of reference alpha images.
 28. Themethod according to claim 26, wherein determining the criteria originalimage and the criteria reference alpha image includes determining one ofthe plurality of original images as the criteria original image bydisplaying selectively the plurality of original images.
 29. The methodaccording to claim 26, which includes preparing the reference alphaimages with respect to the plurality of sequential reference imagesindividually, and summing the sequential reference images to generate acomposite image.
 30. The method according to claim 26, which includesgenerating a string of composite output alpha images by summing thesequential reference images of the output alpha images.
 31. The methodaccording to claim 26, which includes correcting a position of contourof each of the sequential reference images based on the output alphaimages.
 32. A video object clipping apparatus comprising: a storage unitconfigured to store a plurality of sequential original imagesrepresenting a substantially constant movement of a to-be-clipped objectand a plurality of prepared reference alpha images representing aplurality of sequential reference images representing a substantiallyconstant movement of a reference object; a first determination unitconfigured to determine a criteria original image and a criteriareference alpha image from the original images and the reference alphaimages; a second determination unit configured to determine adeformation parameter by deforming the criteria reference alpha image tomatch the criteria original image therewith, the deformation parameterrepresenting at least one of a parallel movement of the original object,a scaling of the original object and a rotation of the original object;a deformation unit configured to deform remaining ones of the referencealpha images according to the determined deformation parameter togenerate output alpha images corresponding to the original images, agenerator to generate a string of the reference alpha images at a framerate in a time axis, a third determination unit configured to determinea set of the criteria original image and the criteria reference alphaimage, a phase matching unit configured to perform a phase matching byequalizing a frame rate between a string of the sequential originalimages and the string of the reference alpha images, the frame ratebeing equalized by downsampling the string of the reference alpha imagesusing the determined criteria alpha image, and a generator to generatethe output alpha images using the string of reference alpha imageshaving the same frame rate as that of the string of the sequentialoriginal images.
 33. The apparatus according to claim 32, wherein thefirst determination unit includes a determination unit configured todetermine one of the plurality of reference alpha images as the criteriareference alpha image by displaying selectively the plurality ofreference alpha images.
 34. The apparatus according to claim 32, whereinthe first determination unit includes a determination unit configured todetermine one of the plurality of original images as the criteriaoriginal image by displaying selectively the plurality of originalimages.
 35. The apparatus according to claim 32, further comprising apreparation unit configured to prepare the reference alpha images withrespect to the plurality of sequential reference images individually,and sum the sequential reference images to generate a composite image.36. The apparatus according to claim 32, further comprising a generatorto generate a string of composite output alpha images by summing thesequential reference images of the output alpha images.
 37. Theapparatus according to claim 32, further comprising a correction unitconfigured to correct a position of contour of each of the sequentialreference images based on the output alpha images.